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ASU Preparatory Academy Website Links 


Homepage 

Arizona State Preparatory Academy http://asuprep.asu.edu 
Elementary Schools 

Phoenix Elementary School http://asuprep.asu.edu/upsp 

Polytechnic Elementary School http://asuprep.asu.edu/pes 
Middle Schools 

Phoenix Middle School http://asuprep.asu.edu/phxms 

Polytechnic Middle School http://asuprep.asu.edu/polyms 
High Schools 

Phoenix High School http://asuprep.asu.edu/phxhs 

Polytechnic High School http://asuprep.asu.edu/polyhs 
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Executive Summary 


In 2013, the Arizona State University (ASU) Preparatory Academy (ASU Prep)? was awarded an Investing 
in Innovation (i3) Grant through the U.S. Department of Education.” The district applied with an 
ambitious grant designed to innovate programming for a total of six schools: two elementary (one 
Kindergarten through grade 5; one Kindergarten through grade 6), two middle (one grades 6 through 8; 
one grades 7 through 8), and two high schools (grades 9 through 12). In 2014, ASU Prep contracted with 
the Educational Policy Institute (EPI) to be the external evaluator of the grant through the programming 
cycle. 


This report represents the final report for this grant and provides data from SY14-15 to SY17-18. 
Included in this report are the final results for the fidelity of implementation and a descriptive analysis of 
the impact data. As a requirement of this grant, EPI assesses the level of fidelity achieved by ASU for the 
five years of the grant. All project data were collected by ASU and provided to EPI for analysis. EPI 
collected additional comparative data from the Arizona Department of Education. 


This report was due to include an exploratory analysis comparing the achievement of the treatment and 
comparison cohorts using statewide assessment data. However, the Arizona Department of Education 
refused to release comparison school data. Thus, these comparisions could not be made. Instead, 
descriptive data on the treatment cohorts are included. Key findings for the final report as noted below. 


ANNUAL PERFORMANCE REPORT (APR) RESULTS 


The percentage of ASU Prep students in grades 3 and 4 who scored proficient in: 


e Reading increased from 45 percent (SY14-15) to 62 percent (SY17-18). 
e Mathematics increased from 48 percent (SY14-15) to 60 percent (SY17-18). 
e Science remained steady at 66 percent (SY14-15 and SY17-18). 


The percentage of ASU Prep students in grades 7 and 8 who scored proficient in: 


e Reading increased from 42 percent (SY14-15) to 47 percent (SY17-18). 
e Mathematics remained steady at 40 percent (SY14-15 and SY17-18). 
e Science increased from 49 percent (SY14-15) to 52 percent (SY17-18). 


1 Formerly University Public Schools, Inc. 
2 see http://www?2.ed.gov/programs/innovation/index.html. 
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In addition: 


e ASU Prep schools achieved a 98 percent graduation rate compared to the Maricopa county rate 
of 78 percent and a state rate of 78 percent in SY16-17 (SY17-18 were not available at the time 
of this reporting). 

e ASU Prep students increased in college readiness as measured by the ACT College Readiness 
Benchmark assessment from 43 percent (SY14-15) to 46 percent (SY17-18). 

e ASU Prep achieved a 99 percent post-secondary enrollment rate in SY17-18. 


Exhibit 1. Change in APR Data from SY14-15 to SY17-18 


APR Data Decrease Same Increase 

G3 and 4 Reading v 
G3 and 4 Mathematics v 
G3 and 4 Science v 

G7 and 8 Reading v 
G7 and 8 Mathematics v 

G7 and 8 Science v 
Graduation Rates v 
ACT College Readiness v 
Post-Secondary Enrollment v 


FIDELITY OF IMPLEMENTATION 
As shown in Exhibit 2 below, ASU Prep met the threshold set by EPI for the Fidelity of Implementation 
assessment. These are the final results for this grant. 


Exhibit 2. Fidelity of Implementation Final Results 


Key Components Activities and Inputs Not Met Met 
Professional Development Staff Development v 
K-12 STEM Focus v 
Learning Lab/STEM Exploratory v 
faction Project Based Learning v 
Technology Based Learning Management System v 
Hybrid Virtual and Full Virtual Learning Opportunities v 
Educational Technology Devices v 
Family Engagement Family Engagement Opportunities NA 
FOI Assessment Final Results v 
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Introduction 


THE ARIZONA STATE UNIVERSITY PREPARATORY ACADEMY I3 PROJECT 

The U.S. Department of Education (ED) awards funding under the Investment in Innovation (i3) program 
as a development grant to explore the effectiveness of a data-driven decision making process that 
incorporates information technology and specific content-focused interventions (i.e., reading, 
mathematics, and science) in grades PK-12. Arizona State University Preparatory Academy (ASU Prep) 
designed Gathering, Reflecting, Owning our Work (GROW) asa five year project to positively influence 
the academic gains of high poverty, largely Hispanic, student populations in and around Phoenix, 
Arizona. ? ASU Prep has identified two elementary schools, two middle schools, and two high schools to 
be evaluated for the duration of this grant. Exhibit 3 notes these six schools.* 


Exhibit 3. Treatment Schools by Name and Grade Levels 


Phoenix Elementary School 90271 K-5 
Polytechnic Elementary School 91308 K-6 
Phoenix Middle School 92326 6-8 
Polytechnic Middle School 91323 7-8 
Phoenix High School 91304 9-12 
Polytechnic High School 91306 9-12 


Project Goals 
GROW provides personalized attention in a university-embedded academic program to prepare 


students for post-secondary success. The guiding objectives of the project are to: 


Close the achievement gap for low socioeconomic students 

Improve student achievement in math 

Improve student achievement in reading 

Improve student achievement in science 

Achieve a graduation rate higher than state averages 

Increase placement of students in STEM-related college majors 

Increase collage and career readiness based on the ACT College Readiness Benchmarks 
Increase family engagement in student learning 


8000 lO) OT 8 NS 


Increase teacher knowledge in integrating technology into core content 


3 The original grant application notes a partnership between University Public Schools, Inc. (UPSI) and Arizona State University (ASU). By 
approval of ED, the name was changed to Arizona State University Preparatory Academy. 
4 ASU Preparatory Academy’s academic year begins in late July and ends in late May of the following year. 
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Theory of Change 
Based upon a four-year theory of change, this project includes three primary components—professional 


development, instruction, and family engagement. The six short-term outcomes (i.e., increased teacher 
knowledge; STEM immersion; create a personalized learning environment; create a ubiquitous 
technology environment; increase family engagement; and connect families and schools) lead to the 
three long-term outcomes of improving student achievement in mathematics, reading, and science. The 
theory of change model is illustrated in Exhibit 4 below. 


Exhibit 4. GROW Theory of Change Logic Model 


Teacher Training. 
¢ Teacher "Cluster" training 90 minutes/ per week 
(ES, MS, and HS) 


Increased teacher knowledge in 
integrating technology into core 
content 


Improved student 
achievement in 
math 


K-12 STEM Focus 
© STEM focused instruction and lab experiences (Grades K-6) 

¢ Students design teaching models to share with younger students 
and their families (Grades 7-8) 

¢ Students utilize ASU Science and Technology labs to support hands- STEM immersion 
on learning (Grades 9-12) (Grades K-12) 


Learning Lab/STEM Exploratory 

¢ Learning Lab 60 minutes/day (Grades K-12) 

¢ STEM Exploratory Class 90 minutes/daily (Polytechnic Middle 
School Grades 7-8) 

Project Based Learning 

¢ Integrate STEM Habits of Mind in the annual Summative Project for 
all students (Grades K-8) 

© Capstone Project (Grades 9-12) 


Improved student 
achievement in reading 


Create a personalized learning 
environment 


Technology-Based Learning Management System 

¢ Learning Management System (LMS) linked to students (Grades K- 
12) unique student portfolio (Grades K-8 utilizes the Cambridge 
International Education Curriculum; Grades 9-12 uses the Board 
Examination System) 


Hybrid Virtual and Full Virtual Learning Opportunities 
e All students will have continuous access to digital learning 
opportunities (Grades K-12) 


Improved student 
achievement in science 


Create a ubiquitous technology 


ji , environment campus-wide 
Educational Technology Devices 


¢ All students are provided with a portable educational technology 
device (Grades K-12) 


Family Engagement Opportunities 

¢ Families participate in the 10-week program offered 
© 30 volunteer hours/ per year on their child's campus 
(Grades K-12) 

¢ Technology closes the communication gap between parents and Connects families and schools 
schools 


Increased family engagement 
in student learning 
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SCOPE OF THE EVALUATION 

The evaluation of the ASU Prep i3 Project entails assessment of the progress on all project goals, 
including annual assessment of progress on all project objectives as reported to ED through Annual 
Performance Review (APR), a Fidelity of Implementation Evaluation, the National Impact Evaluation. 


As shown in Exhibit 5 below, APR data are collected and reported for each year of the grant inclusive of 
the Year 4 (SY17-18). The Option Year 5 (SY18-19) was not awarded. Fidelity of Implementation was 
evaluated during the first two years of the grant and this report provides impact data for the grant. 


The first report (Day 1 of SY14-15), EPI included baseline APR data and baseline impact evaluation data. 
This report (Year 4, SY17-18), includes Fidelity of Implementation and impact data. 


Exhibit 5. Evaluation Summary 


Baseline Confirmatory 


Methods 
A mixed-methods evaluation design was utilized for this evaluation. The aim of the assessment is to 


detail both program implementation and impacts. A description of each form of methodology used is 
detailed in the corresponding sections of the report. 


THIS REPORT 

The evaluation summarizing key components of the ASU Prep i3 grant project performance from 
baseline to the final year, 2017-18. These key components include progress on all performance 
measures listed above and implementation fidelity. This report includes a final review of student 
academic outcomes as well as review of the fidelity of implementation. 
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Academic Data Analysis 


This section details the student outcomes measured over the five years of the grant performance period 
for both annual reporting (internal to ASU Prep School District and external to the U.S. Department of 
Education) and for the National Impact Evaluation. This section addresses the primary goals related to 
student outcomes and is divided accordingly. The six primary goals related to student outcomes are: 


Increase the number of students who perform reading at grade level 
Increase the number of students who perform mathematics at grade level 
Increase the number of students who perform science at grade level 
Achieve a graduation rate higher than state averages 

Increase student college and career readiness 


OY OT Bw NS 


Students enroll in a post-secondary program after graduation 


In the SY13-14, the state of Arizona utilized AIMS (Arizona's Instrument to Measure Standards) as its 
assessment for public school students. In SY14-15 (Year 1), the state of Arizona changed the statewide 
assessment for public school students to the AZMERIT exam for mathematics and reading (i.e. English 
language arts). AIMS continued to be used as the assessment for science (see Exhibit 6). The purpose 
and content of these two tests are very different. AIMS was designed and intended to serve as an exit 
exam for high school graduation and measured the state standards before 2010. AZMERIT, in contrast, 
was designed to measure college-readiness and is not being used as a high school exit exam because the 
content is more difficult.° These differences in content and purpose are illustrated in the decline in test 
scores across the state of Arizona. According to the online publication Arizona Central, part of the USA 
Today network, most students failed the inaugural AZMERIT exam.® 


Exhibit 6. State of Arizona Assessment by Year and Subject 


AIMS AzMERIT AzMERIT AzMERIT AzMERIT 
AIMS AzMERIT AzMERIT AzMERIT AzMERIT 
AIMS AIMS AIMS AIMS AIMS 


In addition to the changes in test content and purpose, AZMERIT also differ from AIMS in its proficiency 


nu nu 


definitions. AIMS defined proficiency levels as “exceeding,” “meeting,” “approaching,” or “falling far 


5 Arizona Central, Arizona Republic; USA Today Network. Accessed March 17, 2016. 
http://www.azcentral.com/story/news/local/arizona/education/2015/11/30/azmerit-scores-most-students-failed-inaugural-test/76561998/ 
® Arizona Central, Arizona Republic; USA Today Network. Accessed March 17, 2016. 
http://www.azcentral.com/story/news/local/arizona/education/2015/11/30/azmerit-scores-most-students-failed-inaugural-test/76561998/ 
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below” proficiency levels. In AZMERIT, students cannot “exceed” proficiency standards but instead, they 
are categorized as “highly proficient.” See Exhibit 7 below for the definitions and categories for AZMERIT 
as compared to AIMS. For ASU Prep’s evaluation, students who scored a “4” or a “3” in reading, 
mathematics, and science were categorized as proficient. Conversely, students who scored a “2” or a “1” 
were categorized as not proficient. 


Exhibit 7. Changes in Testing Proficiency Levels 


4= Exceeds proficiency standards (E) 4= Highly proficient 
3 =Meets proficiency standards (M) 3= Proficient 
2 =Approaching proficiency standards (A) 2= Partially proficient 
1 = Falls far below the proficiency standards (FFB) 1= Minimally proficient 


Quantitative Data Analysis Methodology 
This evaluation utilized the Arizona state assessment AZMERIT for mathematics and reading and AIMS 


science. ”® Because not all grades take the science portion of the test, mathematics is used as a proxy for 
those grades that do not take the science portion in a given year. Grades 4, 8, and high school take the 
AIMS science exam. For all other grades, mathematics scores are used as a proxy for science. EPI and ASU 
Prep recognize the challenges with using a different test and a different subject as a proxy for science. 
When the grant was first awarded, the State of Arizona used AIMS for all subjects. However, the first year 
of the grant, the State changed the assessment. There is no evidence to suggest that the State is 
committed to AZMERIT and will not change assessments again before the conclusion of this grant. EPI will 
continue to monitor these changes as they arise. 


Exhibit 8 shows the grades, state assessment, and academic subjects. 


7 “Students in 3rd through 8th grade will take AZMERIT in English language arts and mathematics at their grade level. In high school, students 
will take AZMERIT end-of-course tests in English language arts and math. The test will be administered to students enrolled in English classes in 
9th through 11th grade, and Algebra |, Geometry, and Algebra II.” Source: https://www.expectmorearizona.org/arizona-aims- 
higher/assessments/faq/ 


8 “AIMS Science is a Standards Based Assessment that measures student proficiency of the Arizona Academic Content Standard in Science. It 


meets federal requirements for student assessment. It is administered in the spring to students in Grades 4, 8, and high school.” Source: 
http://www.azed.gov/assessment/aims-sci/ 
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Exhibit 8. APR Subjects, State Assessments, and Grades 


v v 
v v v v 
AZMERIT AzMERIT 
mathematics used VY mathematics used Y 
as proxy for AIMS as proxy for AIMS 
science science 


INCREASE THE NUMBER OF STUDENTS WHO PERFORM READING AT GRADE LEVEL 

This measure captures the percentage of students in grade 3 and 4 who scored proficient (e.g., a “3” or 
“4”) on the state grade level reading exam. Exhibit 9 shows how Polytechnic and Phoenix students 
performed on the reading portion of the state test administered in a given year. Important to remember 
is that the measures changed from baseline in 2013-14, which used AIMS tests for reading and 
mathematics to AZMERIT. Thus, the baseline figure is not a trusted number for these two categories. 


By the end of the grant period, the percentage of third grade students who scored proficient in reading 
remained similar. Polytechnic students moved from 75 percent in SY14-15 (Year 1) to 70 percent in Year 
4 (SY17-18), while Phoenix students exhibited a slight increase from 28 to 34 percent during the same 
time period. There was a rise in both schools in Year 3, but the percentage deemed proficient decreased 
in the final year. Fourth grade students, however, exhibited large increased from Year 1 to Year 4. The 
percentage of Polytechnic students who were proficient increased from 58 to 79 percent in Year 4, while 
Phoenix students increased from 26 to 54 percent. 


Overall, both schools showed an increase when grades 3 and 4 were combined. The percentage of 
Polytechnic students who scored proficient in reading increased from 66 percent (SY14-15) to 86 
percent (SY17-18). The percentage of Phoenix students who scored proficient in reading increased from 
27 percent (SY14-15) to 40 percent (SY17-18). Assessing both schools and grades together, ASU Prep 
students increased their proficiency in reading from 45 percent (SY14-15) to 62 percent (SY17-18). 
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Exhibit 9. Grades 3 and 4—Proficient in Reading 


Grade 3 


100% 92% 


80% 
60% 
40% 
20% 
0% 
Baseline Year 1 Year 2 Year 3 Year 4 
(SY13-14) (SY14-15) (SY15-16) (SY16-17) (SY17-18) 
AIMS AzMERIT AzMERIT AzMERIT AzMERIT 
=—@—Phoenix ES —@=— Polytechnic ES 
Grade 4 
100% 
87% 85% 84% 
0, 
80% ion a 
60% 
40% 
20% 


0% 


Baseline Year 1 Year 2 Year 3 Year 4 
(SY13-14) (SY14-15) (SY15-16) (SY16-17) (SY17-18) 
AIMS AzMERIT AzMERIT AzMERIT AzMERIT 


—=@=—Phoenix ES —=@=— Polytechnic ES 
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Grade 3 and 4 


100% 


80% 


60% 


40% 


20% 


0% 


Baseline Year 1 Year 2 Year 3 Year 4 
(SY13-14) (SY14-15) (SY15-16) (SY16-17) (SY17-18) 
AIMS AzMERIT AzMERIT AzMERIT AzMERIT 


=@— Phoenix ES —=@=— PolytechnicES —=®—Total 


Exhibit 10 details the percentage of 7‘ and 8"" grade students who scored proficient (e.g., a “3” or “4”) 
on the state grade level mathematics exam. Between Year 1 and Year 4, Polytechnic seventh grade 
students realized a reduction in proficiency from 58 to 54 percent, although the decrease occurred in 
only the last year of the grant. The percentage of Phoenix students who were proficient in reading 
increased from 28 percent to 50 percent. In the eighth grade, Polytechnic students increased from 50 
percent to 60 percent between Year 1 and year 4, compared to an increase for Phoenix students from 
35 to 44 percent. 


When the two grades are aggregated, Polytechnic students moved upwards from 54 to 57 percent in the 
four years and Phoenix students increased from 31 to 47 percent in seventh and eighth grade 
mathematics. Together, the schools moved upwards from 42 percent in Year 1 to 52 percent in Year 4. 
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Exhibit 10. Grades 7 and 8—Proficient in Reading 


Grade 7 
100% 
80% 
60% 
40% 
20% 
0% 
Baseline Year 1 Year 2 Year 3 Year 4 
(SY13-14) (SY14-15) (SY15-16) (SY16-17) (SY17-18) 
AIMS AzMERIT AzMERIT AzMERIT AzMERIT 
=@—Phoenix MS —@= Polytechnic MS 
Grade 8 
100% 
86% 
80% 
60% 
40% 
20% 


0% 


Baseline Year 1 Year 2 Year 3 Year 4 
(SY13-14) (SY14-15) (SY15-16) (SY16-17) (SY17-18) 
AIMS AzMERIT AzMERIT AzMERIT AzMERIT 


=@=—Phoenix MS —=@= Polytechnic MS 
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Grade 7 and 8 


0, 
100% 90% 
0, 
80% 


60% 


40% 


20% 


0% 


Baseline Year 1 Year 2 Year 3 Year 4 
(SY13-14) (SY14-15) (SY15-16) (SY16-17) (SY17-18) 
AIMS AzMERIT AzMERIT AzMERIT AzMERIT 


=@—Phoenix MS —=@=— Polytechnic MS =@®=Total 


INCREASE THE NUMBER OF STUDENTS WHO PERFORM MATHEMATICS AT GRADE 
LEVEL 


Exhibit 11 details the change over time in students proficient in mathematics for grades 3 and 4 cohorts 
for the life of the grant. The percentage of third grade students at Polytechnic increased from 66 to 77 
percent between Year 1 and Year 4, while Phoenix students increased slightly from 43 to 45 percent. 
The percentage of fourth grade students at Polytechnic meeting proficiency in mathematics rose from 
67 to 79 percent, with a high of 93 percent in Year 3. Phoenix students increased from 24 to 42 percent. 


Taking both grades 3 and 4 together, both Polytechnic and Phoenix demonstrated an increase in the 
percentage of students who scored proficient in mathematics. Polytechnic students rose from 66 to 78 
percent, and Phoenix rose from 32 percent to 43 percent. In aggregated form, both schools saw an 
increase from 48 to 60 percent of students meeting proficiency levels in mathematics between Year 1 
and Year 4. 
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Exhibit 11. Grades 3 and 4—Proficient in Mathematics 
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Grade 3 and 4 


100% 94% 


80% 


60% 


40% 


20% 


0% 


Baseline Year 1 Year 2 Year 3 Year 4 
(SY13-14) (SY14-15) (SY15-16) (SY16-17) (SY17-18) 
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—@—Phoenix ES —=@=— PolytechnicES —@Total 


Exhibit 12 details the change over time in students proficient in mathematics for grades 7 and 8 cohorts 
for the grant. The percentage of seventh grade Polytechnic students who met proficiency levels 
decreased from 58 to 49 percent between Year 1 and Year 4. At Phoenix the percentage rose from 26 to 
36 percent. With respect to 8'" grade students, both schools posted increases in proficiencies. The 
percentage of Eighth grade Polytechnic students meeting proficiency in mathematics increased from 47 
to 65 percent, compared to a more modest increase from 32 to 44 percent for Phoenix students. 


With grades seven and eight combined, Polytechnic proficiency in mathematics increased modestly from 
53 to 57 percent and Phoneix students from 29 to 40 percent. The combined proficiency level for both 
grades rose from 40 to 48 percent. 
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Exhibit 12. Grades 7 and 8—Proficient in Mathematics 
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100% 
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80% 
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Grade 7 and 8 
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INCREASE THE NUMBER OF STUDENTS WHO PERFORM SCIENCE AT GRADE LEVEL 
Exhibit 13 details the change over time in students proficient in science for grades 3 and 4 cohorts for 
the grant. Grades 4, 8, and all of high school take the science portion of the AIMS assessment and not 
the AzMERIT science assessment. Because grades 3 and 7 do not take the science exam, the AZMERIT 
mathematics is used as a proxy. Therefore, the charts shown for grades 3 and 7 below are simply the 
mathematics charts repeated. 


Fourth grade students at Phoenix and Polytechnic performed quite differently between Year 1 and Year 
4. At Polytechnic, the percentage of students meeting proficiency in science increased by the second 
year before declining by the fourth year. Polytechnic students started at 80 percent proficiency in 
science, rose to 96 percent in Year 2, and then fell back to 90 percent by Year 4. Students at Phoenix, 
however, had a first-year proficiency rate of 73 percent, but declined to 43 percent in Year 2 before 
rising back to 57 percent in Year 4. Overall, then, Phoenix proficiency rates in science dropped from 73 
to 57 percent between Year 1 and Year 4. 


Taken together, Polytechnic grades 3 and 4 ended up where it started with a 66 percent science 
proficiency rate. 
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Exhibit 13. Grades 3 and 4—Proficient in Science 
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Grade 3 and 4 
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Exhibit 14 details the change over time in students proficient in science for grades 7 and 8 cohorts for 
the grant. Similarly with grades 3 and 4, grade 7 displays the results for the AZMERIT mathematics exam 
and grade 8 shows the results for the AIMS science assessment. 


With regard to the science exam, the proficiency rates for Phoenix and Polytechnic eighth grade 
students remained relatively stable between Year 1 and Year 4. At Polytechnic, the proficiency rating 
rose two points from 81 to 83 percent and the rate at Phoenix rose from 40 to 44 percent. Phoenix did 
jump to 60 percent in Year 2, but that increase was not sustained the following year. 


Combined, the proficiency rate for seventh and eighth grade Polytechnic students decreased from 70 to 
66 percent between Year 1 and Year 4. The Phoenix rates increased from 33 to 40 percent. Aggregated, 
science proficiency rates from the seventh and eighth grades at both schools increased marginally from 
49 to 52 percent. 
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Exhibit 14. Grades 7 and 8—Proficient in Science 


Grade 7 
100% 
89% 
80% 
60% 
40% 
20% 
0% 
Baseline Year 1 Year 2 Year 3 Year 4 
(SY13-14) (SY14-15) (SY15-16) (SY16-17) (SY17-18) 
AIMS AzMERIT AzMERIT AzMERIT AzMERIT 
=@=—Phoenix MS —=@=— Polytechnic MS 
Grade 8 
100% 
85% 83% 


81% 81% 


80% —<$<$< 


60% 


40% 


20% 


0% 
Baseline 
(SY13-14) 
AIMS 


Educational Policy Institute 


Year 1 Year 2 Year 3 
(SY14-15) (SY15-16) (SY16-17) 
AzMERIT AzMERIT AzMERIT 


=@=—Phoenix MS —@= Polytechnic MS 


Year 4 
(SY17-18) 
AzMERIT 


23 


ASU Preparatory Academy i3 Final Report 


Grade 7 and 8 
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ACHIEVE A GRADUATION RATE HIGHER THAN STATE AVERAGES 


Exhibit 15 details the change in graduation rates for the treatment schools as compared to graduation 


rates in Maricopa county’ as a whole and the State of Arizona for Year 2 and Year 3 of the grant. Year 4 


data were not available at the time this report was prepared due to State of Arizona protocol regarding 


data releases.’° As illustrated, the ASU Prep schools had a combined graduation rate of 98 percent in 
2016-17 (Year 3). Students within Maricopa County and at the state level graduated at 78 percent. 


° Representing multiple school districts. Not to be confused with singularly Maricopa Unified School District. 
10 State of Arizona Department of Education Graduation, Dropout & Persistence Rate Technical Manual, (nd), pg. 14. 
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Exhibit 15. Graduation Rates for Target High Schools, County of Maricopa, and State of Arizona Year 2 
and Year 3 (SY16-17) 
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ACT developed a College Readiness Benchmark to help determine how well prepared students are for 
college-level studies. ACT makes these data available to schools by request. Exhibit 16 illustrates that 
students at both Polytechnic and Phoenix saw their readiness scores decrease during Year 2 of the grant 
only to recover by Year 4. Averaged out, the benchmark score increased from 43 percent to 46 percent 
for students in both schools between year 1 and year 4. 
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Exhibit 16. ACT College Readiness Benchmark 
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STUDENTS ENROLL IN A POSTSECONDARY PROGRAM AFTER GRADUATION 

All data regarding post-secondary enrollment are gathered by ASU Prep Guidance Office. This 
performance measure assesses the percentage of students who graduate from one of the treatment 
schools and enrolls in a post-secondary program within six (6) months after graduation. Year 1 
enrollment rates serve as the baseline for this measure. Exhibit 17 shows that ASU Prep schools began 
the program with an average matriculation rate of 91 percent. By Year 3 (SY16-17), almost all students 
matriculated to some form of postsecondary program, iwht a 98 percent rate for Phoneix and 100 
percent for Polytechnic. 


Exhibit 18 provided additional information about matriculation to postsecondary education. By Year 3, 
97 percent of Polytechnic students matriculated to a four-year institution, with 88 percent admitted to 
ASU. Sixty-seven percent of Phoenix students ended up at a four-year institution, with 61 percent 
enrolling at ASU. As well, over 90 percent of seniors at either high school had completed the FAFSA. 
Eighty-three percent of Polytechnic seniors received one or more scholarships, compared to 49 percent 
of Phoenix seniors. 
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Exhibit 17. Enrollment in a Postsecondary Program Six Months after Graduation 
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Exhibit 18. Enrollment in a Postsecondary Program Six Months after Graduation 


Percent of Seniors admitted to ASU* 

Percent of Students Admitted to Any Four Year 

University 69% 97% 
Percent of Students Admitted to Any Four or Two Year 

College or University 69% 100% 
Percent Seniors Admitted to Any Postsecondary 98% 100% 
Percent of Seniors who have completed FAFSA 91% 93% 
Percent Seniors with one or more scholarships 49% 83% 
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Fidelity of Implementation 


The ASU Prep set ambitious goals through this i3 grant project. As mentioned earlier in this report, at 
the outset, the district worked closely with the evaluation teams to ensure their ability to track fidelity 
for implementation of the grant in order to effectively achieve the goals. As outlined above, the district’s 
ability to impact student academic outcomes has been a major success of the grant implementation. To 
implement the grant, the district set nine (9) guiding project objectives to achieve these positive student 
outcomes. 


The guiding objectives of the project are to: 


Close the achievement gap for low socioeconomic students 

Improve student achievement in math 

Improve student achievement in reading 

Improve student achievement in science 

Achieve a graduation rate higher than state averages 

Increase placement of students in STEM-related college majors 

Increase collage and career readiness based on the ACT College Readiness Benchmarks 
Increase family engagement in student learning 


$0 00 2 OY OU ES oO SID AE 


Increase teacher knowledge in integrating technology into core content 


A Fidelity of Implementation matrix was prepared by the evaluation team and ASU Prep district to assess 
progress toward these overarching goals (see Appendix). The assessment was divided into three major 
components: 1) professional development, 2) instruction, and 3) family engagement. This section details 
the specific implementation of the grant according to the nine (9) objectives and the three (3) fidelity of 
implementation components. 


Qualitative Data Analysis 
At the outset of the grant, ASU Prep has worked closely with the external evaluator, EPI, to develop a set 


of measures and methods to assess the fidelity of grant implementation and achievement of all project 
goals (outlined above). ASU Prep aimed to use the i3 grant to enhance district processes, strengthen the 
teaching base and educational performance, in order to strengthen student outcomes. While the 
majority of data collected and assessed to address overall performance measures and the national 
impact evaluation entail an analysis of quantitative data only, a mixture of qualitative and quantitative 
data was collected to assess the fidelity of implementation to the grant. This includes data regarding 
program implementation such as professional development training outlines, examples of curriculum 
developed under the project and survey results. All fidelity data is provided by ASU Prep through a 
secure online account to house and share the implementation data with the evaluators. Details 
regarding individual data reviewed for the fidelity of implementation is included later in this report. 
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ACCOMPLISHED PROJECT OBJECTIVES 

Fidelity toward goals were assessed through three major components: 1) professional development, 2) 
instruction, and 3) family engagement. The fidelity matrix was developed prior to the implementation of 
the program and used to assess Year 1 (SY14-15) and assessed for Year 2 (SY15-16). Exhibit 19 below 
highlights accomplishments from each of the three assessed components. ASU Prep met the fidelity 
threshold level for this project for both years that fidelity was measured. 


Exhibit 19. Implementation of Project Objectives 


PROFESSIONAL DEVELOPMENT 
Teacher "Cluster" Learning 1=Met 1 1 
O= Not Met 
Staff Development Management Systems (LMS) ot Me 
training 90 minutes/ per week(ES, 
MS, and HS) pau ‘ 1 
’ O= Not Met 
INSTRUCTION 
STEM focused instruction and lab 1=Met 1 1 
experiences (Grades K-6) O= Not Met 
K-12 STEM Focus | Design teaching models for younger 1=Met 1 1 
students (Grades 7-8) O= Not Met 
Utilize ASU Science and Technology 1=Met 1 1 
labs (Grades 9-12) O= Not Met 
Learning Lab 60 minutes/day 1=Met 1 1 
(Grades K-12) O= Not Met 
Learning Lab/ 
STEM Exploratory STEM Exploratory Class 90 
; : nage 1=Met 
minutes/daily (Polytechnic Middle O= Not Met 1 1 
School Grades 7-8) 7 
Integrate STEM Habits of Mind (Six 
STEM Proficiencies) in the annual 1=Met 
: : 1 1 
7 Summative Project for all students O= Not Met 
Project Based 
; (Grades K-8) 
Learning 
. 1=Met 
Capstone Project (Grades 9-12) O= Not Met al Hl 
Pr rosie Learning Management System 1=Met 0 0 
6 8 (LMS) (Grades K-12) O= Not Met 
System 
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Hybrid Virtual and Continuous access to digital 1=Met 
Full Virtual learning opportunities (Grades K- O- at Met 1 1 
Learning Opportunities 12) 
Educational 1 per classroom Educational 1=Met 1 1 
Technology Devices | Technology Devices(Grades K-12) O= Not Met 


Volunteer Commitment- 30 : 
. Monitored but 
volunteer hours/ per year on their NA NA 
aa not assessed 
Family Engagement child's campus (Grades K-12) 
Opportunities** 


ee Monitored but 
Parent and school communication NA NA 
not assessed 


*This measure was pushed to Year 2 and Year 3 due to delays in purchasing. ASU’s ED contact was informed of this adjustment. 
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Appendix 1. Fidelity of Implementation Plan 


Teacher 
"Cluster" 
Learning 
Management 
Systems (LMS) 
training 90 
minutes/ per 
week(ES, MS, 
and HS) 
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Teachers will gain knowledge on 
the integration of technology 
into the curriculum, particularly 
as it impacts teaching and 
instruction, pedagogical 
innovation, the interpretation of 
curriculum goals and objectives, 
and assessment practices. 


80 percent of Grades 4-12 
teachers will attend 80 
percent of the training 90 
minutes/per week 


1=Met 
O= Not 
Met 


Quarterly attendance 
records (one per semester 
for each grade Grades 4- 
12) 


September (Q1), 
December (Q2), March 
(Q3), June (Q4) 


(Years 1 and 2 of the 
grant) 


10 percent of 
participating teachers 
(Grades 4-12) given pre 
and post surveys 
(averaging over 80 
percent agreement in 
knowledge gains in 
technology integration 
into the curriculum) 


1=Met 
O= Not 
Met 


Annual pre and post 
survey data (Grades 4-12) 


Pre- Survey collection in 
August /September 


Post- survey collection 
in May /June 


(Years 1 and 2 of the 
grant) 


Fidelity Threshold Score 


Met=>1 
Not Met=0 
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Learning Lab Students work independently 35 . September (Q1), 
60 with a personalized Play List of percent of students 1=Met Quarterly access records December (Q2), March 
fnintites!day educational games, skill (Grades K-12) participate O= Not (one per classroom per (Q3), June (Q4) 
(Grades K-12) builders, and test preparation in Learning Lab Met Grade 4-12) 
Feathing tabs tools in the school computer lab (Years 1 a of the 
STEM and a highly qualified teacher 8 
STEM prepared to deliver one-on-one 
Exploratory : ‘ 
Exploratory instruction as necessary. Each 80 percent of students Aibe 
Class 90 Play List is assigned by (Grades 7-8) will 1=Met Annual attendance 
minutes/daily classroom teachers, based on participate in STEM O= Not records (one per (years tanaperthe 
(Polytechnic the Learning Management Exploratory Class for 90 Met classroom per Grade 7-8) 
Middle School System recommendations. minutes/daily grant) 
Grades 7-8) 
Integrate 
STEM Habits Quarterly grading Records 
of Mind (Six 80 percent of students (one per classroom per September (Q1), 
STEM Students will learn how to build (Grades K-8) will 1=Met Grade 4-8) December (Q2), March 
Proficiencies) tech apps to meet specific demonstrate at least two be het (Q3), June (Q4) 
in the annual needs, and will train others to STEM proficiencies in Ret Quarterly STEM 
Summative use them through Tech Prep U. Summatives and Proficiency Rubrics - (Years 1 and 2 of the 
Project for all Capstones Content analysis (one per grant) 
students Grade 4-8 per school) 
Project Based (Grades K-8) 
Learning High school students will pursue 
an authentic learning 
experience demonstrating 
Capstone content from all core subjects 80 percent of students fewer Annual Canciones Prolee June 
. with opportunities to receive (Grades 9-12) will G 
Project : O= Not Grading records (one per 
(Grades 9-12) guidance from ASU graduate complete a Capstone ran slassroarn per Grade 0°19) (Years 1 and 2 of the 


students, faculty, and industry 

experts as appropriate to their 

selected Capstone topic, both in 
person and virtually. 


Project 


grant) 
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K-12 STEM 
Focus 


Students will receive STEM- 


STEM focused focused instruction K-6, with September (Q1), 
: : classroom instruction and 100 percent of students Walk through protocol December (Q2), March 
instruction f ‘ : ‘ 1=Met 
aaah authentic lab experiences. will receive STEM focused O= Not (one per Grade 4-6per (Q3), June (Q4) 
: Students will explore concepts instruction weekly 2 treatment school for 
experiences : 3 Met ; F 
and careers in sciences, (Grades K-6) mathematics and science) (Years 1 and 2 of the 
(Grades K-6) : : 
technology, engineering and grant) 
mathematics. 
I ition to STEM f 
ae | een spate, 
teaching i : 80 percent of STEM , December (Q2), March 
develop their own teaching 1=Met Quarterly grading records 
models for Academy students (Grade (Q3), June (Q4) 
modules to share core : O= Not one per classroom (Grades 
younger ; . 7 and Grade 8) will 
curriculum concepts with : Met 7-8) 
students ; develop a teaching model (Years 1 and 2 of the 
younger students and their 
(Grades 7-8) ia grant) 
families. 
2 In high school, in addition to 
palit pues STEM focused instruction, UPSI | 80 percent of Polytechnic se rreIneL UT, 
Science and 7 1=Met Quarterly attendance December (Q2), March 
students utilize the ASU state of students (Grades 9-12) 
Technology , Bi ea bs O= Not Records (one per (Q3), June (Q4) 
the art Science and Technology will utilize the ASU lab 
labs (Grades iehetesunnbre hands en es Met classroom per Grade 9-12) 
9-12) PP Y (Years 1 and 2 of the 


learning. 


grant) 


Educational Policy Institute 


33 


ASU Preparatory Academy i3 Final Report 


The program will be linked to 
unique student portfolios that 
follow students throughout their 


Technology Learning Saucatlehal fares ts atid . Data Dashboard will be Annual content review of 
Based subsequent individual learning : : June 
: Management ._ {functional and demonstrate 1=Met Data Dashboard site (one 
Learning plans. (For Grades K-8, the LMS is : : (Years 1 and 2 of the 
System (LMS) . an alignment to the grade | O= Not Met | evaluation per Grade 4-12 
Management aligned to the framework of the . grant) 
(Grades K-12) ; : appropriate framework per treatment school) 
System Cambridge International 
Curriculum; high school students 
will be monitored according to the 
Board Examination System) 
Hybrid Virtual} Continuous 80 percent of classrooms septembe (00): 
fa : Sigh osk December (Q2), March 
and] access to digital |Students will be able to access and} (Grades K-12) will utilize ‘eKaee Quarterly access records (Q3), June (Q4) 
Full Virtual learning integrate technology into any digital learning O= ast ie (one per class per Grade 4-12 : 
Learning| opportunities subject in the classroom. opportunities for classroom} ~~ per semester) 
wt : : (Years 1 and 2 of the 
Opportunities| (Grades K-12) instruction (Frequency?) eran) 
1 per classroom | Each classroom will be provided a 
Educational} Educational portable device for student use, | 80 percent of classrooms jie Annual distribution records June 
Technology} Technology bringing new on-the-go (Grades K-12) have received O= eee (one per classroom per (Years 1 and 2 of the 
Devices Devices applications featuring video and a portable device ~ Grade 4-12) grant) 
(Grades K-12) | games to build concept fluency. 
Met =>8 
Fidelity Threshol 
idelity Threshold Score Not Met=<7 


Educational Policy Institute 


34 


ASU Preparatory Academy i3 Final Report 


Volunteer 
Commitment- 
30 volunteer 
hours/ per year 
on their child's 


Families can monitor their 
movement towards their annual 
required commitment of 
volunteer hours on their child’s 
campus. 


campus 
(Grades K-12) 


Families complete the 
required 30 hours of 
volunteerism on their 
students campus: 
Yr1: 30 percent 
Yr2: 35 percent 
Yr3: 40 percent 
Yr4: 45 percent 
Yr5: 50 percent 


Quarterly parent 
volunteer records (one 
per classroom per Grade 
4-12) 


September (Q1), 
December (Q2), March 
(Q3), June (Q4) 


(Years 1 and 2 of the 
grant) 


Family 
Engagement 
Opportunities 


Parent and 
school 
communication 


Technology closes the 
communication gap between 
parents and schools 


Fidelity Threshold Score 


90 parents in total from 
the treatment schools 
(n=5 from each grade for 
each treatment 
school)for each 
treatment school will be 
given a pre and post 
surveys; 


3 administrators (one 
from one elementary 
school, one from one 
middle school; one from 
one high school) will be 
given pre and post 
surveys 
Approval scores will 
average at least 75 
percent each year 


Met=>1 
Not Met=0 


Annual Pre and Post 
Surveys 


Pre- Survey collection in 
August /September 


Post- survey collection 
in May /June 
(Years 1 and 2 of the 
grant) 
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